Full-color visibility model using CSF which varies spatially with local luminance

ABSTRACT

The present disclosure relate generally to image signal processing, color science and signal encoding. Signal encoding can be applied to color image data through use of a luminance contrast sensitivity function and a chrominance contrast sensitive function. Of course, other features, combinations and claims are disclosed as well.

RELATED APPLICATION DATA

This patent application is a continuation of U.S. patent applicationSer. No. 15/426,630, filed Feb. 7, 2017 (now U.S. Pat. No. 9,805,435)which is a continuation of U.S. patent application Ser. No. 15/137,401,filed Apr. 25, 2016 (now U.S. Pat. No. 9,565,335), which is acontinuation in part of U.S. patent application Ser. No. 14/588,636,filed Jan. 2, 2015 (now U.S. Pat. No. 9,401,001), which claims thebenefit of U.S. Provisional Patent Application No. 61/923,060, filedJan. 2, 2014. U.S. patent application Ser. No. 15/137,401 also claimsthe benefit of U.S. Provisional Patent Application No. 62/152,745, filedApr. 24, 2015. Each of these patent documents are hereby incorporatedherein by reference in its entirety.

This application is related to U.S. Pat. Nos. 9,224,184, 9,129,277 and8,199,969; US Published Patent Application Nos. US 2010-0150434 A1, US2014-0119593 A1, US 2015-0156369 A1; and U.S. patent application Ser.No. 13/975,919, filed Aug. 26, 2013, now U.S. Pat. No. 9,449,357.

The above patent documents are each hereby incorporated herein byreference in their entirety.

TECHNICAL FIELD

The present disclosure relates generally to color science, imageprocessing, steganographic data hiding and digital watermarking.

BACKGROUND AND SUMMARY

The term “steganography” generally means data hiding. One form of datahiding is digital watermarking. Digital watermarking is a process formodifying media content to embed a machine-readable (ormachine-detectable) signal or code into the media content. For thepurposes of this application, the data may be modified such that theembedded code or signal is imperceptible or nearly imperceptible to auser, yet may be detected through an automated detection process. Mostcommonly, digital watermarking is applied to media content such asimages, audio signals, and video signals.

Digital watermarking systems may include two primary components: anembedding component that embeds a watermark in media content, and areading component that detects and reads an embedded watermark. Theembedding component (or “embedder” or “encoder”) may embed a watermarkby altering data samples representing the media content in the spatial,temporal or some other domain (e.g., Fourier, Discrete Cosine or Wavelettransform domains). The reading component (or “reader” or “decoder”)analyzes target content to detect whether a watermark is present. Inapplications where the watermark encodes information (e.g., a message orpayload), the reader may extract this information from a detectedwatermark.

A watermark embedding process may convert a message, signal or payloadinto a watermark signal. The embedding process then combines thewatermark signal with media content and possibly another signals (e.g.,an orientation pattern or synchronization signal) to create watermarkedmedia content. The process of combining the watermark signal with themedia content may be a linear or non-linear function. The watermarksignal may be applied by modulating or altering signal samples in aspatial, temporal or some other transform domain.

A watermark encoder may analyze and selectively adjust media content togive it attributes that correspond to the desired message symbol orsymbols to be encoded. There are many signal attributes that may encodea message symbol, such as a positive or negative polarity of signalsamples or a set of samples, a given parity (odd or even), a givendifference value or polarity of the difference between signal samples(e.g., a difference between selected spatial intensity values ortransform coefficients), a given distance value between watermarks, agiven phase or phase offset between different watermark components, amodulation of the phase of the host signal, a modulation of frequencycoefficients of the host signal, a given frequency pattern, a givenquantizer (e.g., in Quantization Index Modulation) etc.

The present assignee's work in steganography, data hiding and digitalwatermarking is reflected, e.g., in U.S. Pat. Nos. 6,947,571; 6,912,295;6,891,959. 6,763,123; 6,718,046; 6,614,914; 6,590,996; 6,408,082;6,122,403 and 5,862,260, and in published specifications WO 9953428 andWO 0007356 (corresponding to U.S. Pat. Nos. 6,449,377 and 6,345,104).Each of these patent documents is hereby incorporated by referenceherein in its entirety. Of course, a great many other approaches arefamiliar to those skilled in the art. The artisan is presumed to befamiliar with a full range of literature concerning steganography, datahiding and digital watermarking.

One possible combination of the inventive teaching is a methodincluding: receiving a color image or video; transforming the colorimage or video signal by separating the color image or video into atleast first data representing a first color channel of the color imageor video and second data representing a second color channel of thecolor image or video, where the first data comprises a digital watermarksignal embedded therein and the second data comprises the digitalwatermark signal embedded therein with a signal polarity that isinversely related to the polarity of the digital watermark signal in thefirst data; subtracting the second data from the first data to yieldthird data; using at least a processor or electronic processingcircuitry, analyzing the third data to detect the digital watermarksignal; once detected, providing information associated with the digitalwatermark signal.

Another combination is a method including: obtaining first datarepresenting a first chrominance channel of a color image or video,where the first data comprises a watermark signal embedded therein;obtaining second data representing a second chrominance channel of thecolor image or video, the second data comprising the watermark signalembedded therein but with a signal polarity that is inversely related tothe polarity of the watermark signal in the first data; combining thesecond data with the first data in manner that reduces image or videointerference relative to the watermark signal, said act of combiningyielding third data; using at least a processor or electronic processingcircuitry, processing the third data to obtain the watermark signal;once obtained, providing information associated with the watermarksignal.

Still another combination is an apparatus comprising: a processor orelectronic processing circuitry to control: (a) handling of first datarepresenting a first color channel of a color image or video, where thefirst data comprises a watermark signal embedded therein; (b) handlingof second data representing a second color channel of the color image orvideo, the second data comprising the watermark signal embedded thereinbut with a signal polarity that is inversely related to the polarity ofthe watermark signal in the first data; (c) combining the second datawith the first data in manner that reduces image or video interferencerelative to the watermark signal, the combining yielding third data; (d)processing the third data to obtain the watermark signal; and (e) onceobtained, providing information associated with the watermark signal.

Yet another possible combination is a method including: a methodincluding: obtaining first data representing a first chrominance channelof a color image or video signal; obtaining second data representing asecond chrominance channel of the color image or video signal; using aprocessor or electronic processing circuitry, embedding a watermarksignal in the first data with a first signal polarity; using a processoror electronic processing circuitry, transforming the second data byembedding the watermark signal in the second data so that when embeddedin the second data the watermark signal comprises a second signalpolarity that is inversely related to the first signal polarity of thewatermark signal in the first data; combining the watermarked first dataand the watermarked second data to yield a watermarked version of thecolor image or video signal, whereby during detection of the watermarksignal from the watermarked version of the color image or video signal,the second data is combined with the first data in a manner that reducesimage or video signal interference relative to the watermark signal.

Still a further combination is a digital watermarking method comprising:using a programmed electronic processor, modeling a first color ink anda second color ink in terms of CIE Lab values; modulating the valueswith a watermarking signal; scaling the modulated values in a spatialfrequency domain; spatially masking the scaled, modulated values;providing the spatially masked, scaled, modulated values, such valuescarrying the watermark signal.

Another combination includes an apparatus, comprising: memory forstoring: i) a luminance contrast sensitivity function (CSF1), ii) achrominance contrast sensitivity function (CSF2), and iii) datarepresenting color imagery; and one or more processors configured for:applying the CSF1 and the CSF2 to predict degradation of image areasassociated with an application of digital watermarking to the datarepresenting color imagery, in which the CSF1 varies depending onluminance values associated with local regions of the data representingcolor imagery and in which the CSF1 is used for processing luminancedata and the CSF2 is used for processing chrominance data; transformingthe data representing color imagery with digital watermark, in which thedigital watermarking is guided based on results obtained from theapplying including predicted degradation of image areas.

In one implementation the CSF1 varies spatially, perhaps in terms ofspatial width. In another implementation, the CSF2 varies spatially interms of spatial width.

The CSF1 may be applied to predict degradation of image areas producesimage blurring as the predicted degradation, in which the CSF1 varies sothat relatively more blurring occurs as luminance of a local imageregion decreases.

In some implementations the digital watermarking is guided based onresults obtained from the applying by varying embedding strength acrossdifferent image areas of the data representing color imagery based onpredicted degradation of the different image areas. The predicteddegradation of the digital watermarking across the different image areasmay include uniform predicted degradation.

The one or more processors may be configured for processing the datarepresenting color imagery with an attention model to predict visualtraffic areas.

In other implementations the digital watermarking may be guided based onthe results obtained from the predicted visual traffic areas and thepredicted degradation of image areas.

In some cases, the chrominance contrast sensitivity function (CSF2)includes a blue-yellow contrast sensitivity function and a red-greencontrast sensitivity function.

The CSF2 may vary depending on luminance values associated with localregions of the obtained color image data.

The transforming the data representing color imagery with digitalwatermarking may embed a machine-readable code into the datarepresenting color imagery. And, in some cases, the imagery comprisesvideo.

Further combinations, aspects, implementations, features and advantageswill become even more apparent with reference to the following detaileddescription and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 represents a color image.

FIG. 2 represents a first color channel (‘a’ channel) of the color imagerepresentation shown in FIG. 1.

FIG. 3 represents a second color channel (‘b’ channel) of the colorimage representation shown in FIG. 1.

FIG. 4 is a representation of the sum of the first color channel of FIG.2 and the second color channel of FIG. 3 (e.g., a+b).

FIG. 5 is a graph showing a histogram standard deviation of FIG. 4.

FIG. 6 is a representation of the difference between the first colorchannel of FIG. 2 and the second color channel of FIG. 3 (a−b).

FIG. 7 is a graph showing a histogram standard deviation of FIG. 6.

FIG. 8 is an image representation of the difference between the firstcolor channel of FIG. 2 (including a watermark signal embedded therein)and the second color channel of FIG. 3 (including the watermark signalembedded therein).

FIG. 9 is a graph showing a histogram standard deviation of FIG. 8.

FIGS. 10A and 10B are block diagrams showing, respectively, an embeddingprocess and a detection process.

FIG. 11 is a diagram showing watermarks embedded in first and secondvideo frames.

FIG. 12 is a diagram showing a detailed signal size view with inkincrements of 2%, and the addition of press visibility constraints.

FIG. 13A corresponds to Appendix D's FIG. 1, which shows a quality rulerincreasing in degradation from B (slight) to F (strong).

FIG. 13B corresponds to Appendix D's FIG. 2, which shows thumbnails ofthe 20 color patch samples with a watermark applied.

FIG. 14 corresponds to Appendix D's FIG. 3, which shows a mean observerresponses with 95% confidence intervals for color patches.

FIG. 15 corresponds to Appendix D's FIG. 4, which shows mean observerresponse compared with a proposed visibility model.

FIG. 16 corresponds to Appendix D's FIG. 5, which shows mean observerresponse compared with the proposed visibility model with luminanceadjustment.

FIG. 17 corresponds to Appendix D's FIG. 6, which shows mean observerresponse compared with S-CIELAB.

FIG. 18 corresponds to Appendix D's FIG. 7, which shows watermarkembedding with uniform signal strength (left) and equal visibility froma visibility model (right). The insets are magnified to show imagedetail.

FIG. 19 corresponds to Appendix D's FIG. 8, which shows visibility mapfrom uniform signal strength embedding (left) and equal visibilityembedding (right) from FIG. 18.

FIG. 20 corresponds with Appendix D's FIG. 9, which shows Apple tart,Giraffe stack and Pizza puff design used in tests.

FIG. 21 is a block diagram for a color visibility model.

FIG. 22 is a block diagram for a color visibility model with CSFs variedaccording to image local luminance.

FIG. 23A is a block diagram for a color visibility model to achieveequal visibility embedding (EVE).

FIG. 23B is a diagram showing EVE embedding compared to uniformembedding.

DETAILED DESCRIPTION

Portions of the following disclosure discusses a digital watermarkingtechnique that utilizes at least two chrominance channels (also called“color planes,” “color channels” and/or “color direction”). Chrominanceis generally understood to include information, data or signalsrepresenting color components of an image or video. In contrast to acolor image or video, a grayscale (monochrome) image or video has achrominance value of zero.

Media content that includes a color image (or color video) isrepresented in FIG. 1. An industry standard luminance and chrominancecolor space is called “Lab” (for Lightness (or luminance), plus ‘a’ and‘b’ color channels) that can be used to separate components of imagesand video. FIG. 2 is an ‘a’ channel representation of FIG. 1 (shown ingrayscale), and FIG. 3 is a ‘b’ channel representation of FIG. 1 (shownin grayscale). Of course, our inventive methods and apparatus will applyto and work with other color schemes and techniques as well. Forexample, alternative luminance and chrominance color schemes include“Yuv” (Y=luma, and ‘u’ and ‘v’ represent chrominance channels) and“Ycc.” (also a dual chrominance space representation).

Let's first discuss the additive and subtractive effects on FIGS. 2 and3. FIG. 4 illustrates a representation of the result of adding the ‘a’channel (FIG. 2) with the ‘b’ channel (FIG. 3). FIG. 6 illustrates arepresentation of the result of subtracting the ‘b’ channel (FIG. 3)from the ‘a’ channel (FIG. 2). The result of subtracting the ‘b’ channelfrom the ‘a’ channel yields reduced image content relative to adding thetwo channels since the ‘a’ and ‘b’ color planes have correlated imagedata in the Lab scheme. (In typical natural imagery, the ‘a’ and ‘b’chrominance channels tend to be correlated. That is to say where ‘a’increases, ‘b’ also tends to increase. One measure of this is to measurethe histogram of the two chrominance planes when they are added (seeFIG. 5), and compare that to the histogram when the two color planes aresubtracted (see FIG. 7). The fact that the standard deviation of FIG. 7is about half that of FIG. 5 also supports this conclusion, andillustrates the reduction in image content when ‘b’ is subtracted from‘a’) In this regard, FIG. 4 provides enhanced or emphasized imagecontent due to the correlation. Said another way, the subtraction of theFIG. 3 image from FIG. 2 image provides less image interference orreduces image content. The histogram representations of FIG. 4 and FIG.6 (shown in FIGS. 5 and 7, respectively) further support thisconclusion.

Now let's consider watermarking in the context of FIGS. 2 and 3.

In a case where a media signal includes (or may be broken into) at leasttwo chrominance channels, a watermark embedder may insert digitalwatermarking in both the ‘a’ color direction (FIG. 2) and ‘b’ colordirection (FIG. 3). This embedding can be preformed in parallel (ifusing two or more encoders) or serial (if using one encoder). Thewatermark embedder may vary the gain (or signal strength) of thewatermark signal in the ‘a’ and ‘b’ channel to achieve improved hidingof the watermark signal. For example, the ‘a’ channel may have awatermark signal embedded with signal strength that greater or less thanthe watermark signal in the ‘b’ channel. Alternatively, the watermarksignal may be embedded with the same strength in both the ‘a’ and ‘b’channels. Regardless of the watermark embedding strength, watermarksignal polarity is preferably inverted in the ‘b’ color plane relativeto the ‘a’ color plane. The inverted signal polarity is represented by aminus (“−”) sign in equations 1 and 2.WMa=a(channel)+wm  (1)WMb=b(channel)−wm  (2)

WMa is a watermarked ‘a’ channel, WMb is a watermarked ‘b’ channel, andwm represents a watermark signal. A watermarked color image (including Land WMb and WMa) can be provided, e.g., for printing, digital transferor viewing.

An embedded color image is obtained (from optical scan data, memory,transmission channel, etc.), and data representing the color image iscommunicated to a watermark detector for analysis. The detector (or aprocess, processor or electronic processing circuitry used inconjunction with the detector) subtracts WMb from WMa resulting in WMresas shown below:WMres=WMa−WMb  (3)WMres=(a+wm)−(b−wm)  (4)WMres=(a−b)+2*wm  (5)

This subtraction operation yields reduced image content (e.g., FIG. 6)as discussed above. The subtraction or inverting operation of the colorchannels also emphasizes or increases the watermark signal (2*wm),producing a stronger watermark signal for watermark detection. Indeed,subtracting the color channels increases the watermark signal-to-mediacontent ratio: WMres=(a−b)+2*wm.

FIG. 8 illustrates the result of equation 5 (with respect to watermarkedversions of FIG. 2 and FIG. 3). As shown, the perceptual “graininess” or“noise” in the image corresponds to the emphasized watermark signal. Theimage content is also reduced in FIG. 8. A histogram representation ofFIG. 8 is shown in FIG. 9 and illustrates a favorable reduction of imagecontent.

A watermark detector may extract or utilize characteristics associatedwith a synchronization signal (if present) from a frequency domainrepresentation of WMres. The detector may then use this synchronizationsignal to resolve scale, orientation, and origin of the watermarksignal. The detector may then detect the watermark signal and obtain anymessage or payload carried thereby.

To even further illustrate the effects of improving the watermarksignal-to-media content ratio with our inventive processes and systems,we provide some additive and subtractive examples in the content ofwatermarking.

For the following example, a watermark signal with the same polarity isembedded in each of the ‘a’ color channel and the ‘b’ color channel. Thesame signal polarity is represented by a plus (“+”) sign in equations 6and 7.WMa=a+wm  (6)WMb=b+wm  (7)

WMa is a watermarked ‘a’ channel, WMb is a watermarked ‘b’ channel, andwm represents a watermark signal. A watermarked color image (including Land WMb and WMa) can be provided, e.g., for printing, digital transferor viewing.

An embedded color image is obtained, and data representing the colorimage is communicated to a watermarked detector for analysis. Thedetector (or a process, processor, or electronic processing circuitryused in conjunction with the detector) adds the ‘a’ and ‘b’ colorchannels to one another (resulting in WMres) as shown below:WMres=WMa+WMb  (8)WMres=(a+wm)+(b+wm)  (9)WMres=(a+b)+2*wm  (10)

This addition operation results in increased image content (e.g., FIG.4). Indeed, image interference during watermark detection will begreater since the two correlated ‘a’ and ‘b’ color channels tend toreinforce each other.

By way of further example, if WMb is subtracted from WMa (with watermarksignals having the same polarity), the following results:WMres=WMa−WMb  (11)WMres=(a+wm)−(b+wm)  (12)WMres=(a−b)+≈0*wm  (13)

A subtraction or inverting operation in a case where a watermark signalincludes the same polarity decreases image content (e.g., FIG. 4), butalso significantly decreases the watermark signal. This may result inpoor—if any—watermark detection.

FIGS. 10A and 10B are flow diagrams illustrating some related processesand methods. These processes may be carried out, e.g., via a computerprocessor, electronic processing circuitry, printer, handheld devicesuch as a smart cell phone, etc.

With reference to FIG. 10A, a color image (or video) is obtained andseparated into at least two (2) color channels or planes (10). Awatermark signal is determined for the color image or video (12). Ofcourse, the watermark signal for the color image or video may bedetermined prior to or after color plane separation. The determinedwatermark signal is embedded in a first of the color planes (14). Aninverse polarity version of the watermark signal is embedded in a secondcolor plane. The color planes are recombined (perhaps with datarepresenting luminance) to form a composite color image.

With reference to FIG. 10B, a watermarked color image or video isobtained or received (11). The color image (or video) has or can beseparated into at least two (2) color planes or channels (13). A firstcolor plane includes a watermark signal embedded therein. A second colorplane includes the watermark signal embedded therein with a polaritythat is inversely related to the watermark signal in the first colorplane. The watermarked second color plane is subtracted from thewatermarked first color (15). The result of the subtraction is analyzedto detect the watermark signal. A detected watermark message, signal orpayload can be provided (19), e.g., to a remote database to obtainrelated metadata or information, to a local processor, for display, to arights management system, to facilitate an online transaction, etc.

In addition to the Lab color scheme discussed above, a watermark signalmay be embedded in color image (or video) data represented by RGB, Yuv,Ycc, CMYK or other color schemes, with, e.g., a watermark signalinserted in a first chrominance direction (e.g., red/green direction,similar to that discussed above for the ‘a’ channel) and a secondchrominance direction (e.g., a blue/yellow direction, similar to thatdiscussed above for the ‘b’ channel). For watermark signal detectionwith an alternative color space, e.g., an RGB or CMYK color space, animage can be converted to Lab (or other color space), or appropriateweights of, e.g., RGB or CMY channels, can be used. For example, thefollowing RGB weights may be used to calculate ‘a’−‘b’: ChrominanceDifference=0.35*R−1.05*G+0.70*B+128, where R, G and B are 8-bitintegers.

Further Considerations of Video

The human contrast sensitivity function curve shape with temporalfrequency (e.g., relative to time) has a very similar shape to thecontrast sensitivity with spatial frequency.

Successive frames in a video are typically cycled at about at least 60Hz to avoid objectionable visual flicker. So-called “flicker” is due tothe high sensitivity of the human visual system (HVS) to high temporalfrequency changes in luminance. The human eye is about ten (10) timesless sensitive to high temporal frequency chrominance changes.

Consider a video sequence with frames as shown in FIG. 11. A chrominancewatermark can be added to frame 1 per the above description for images.In a similar way, a watermark is added to frame 2 but the polarity isinverted as shown in FIG. 11.

In order to recover the watermark, pairs of frames are processed by awatermark detector, and the ‘a’ channels are subtracted from each otheras shown below.Det_a=(a1+wm)−(a2−wm)=(a1−a2)+2*wm  (14)

Det_a refers to watermark detection processing of the ‘a’ channel.Because of the temporal correlation between frames, the image content inequation 14 is reduced while the watermark signal is reinforced.

In a similar way the ‘b’ channels are also subtracted from each otherDet_b=(b1−wm)−(b2+wm)=(b1−b2)−2*wm  (15)

Det_a refers to watermark detection processing of the ‘b’ channel.Equation 14 and 15 are then subtracted from each other as shown below inequation 16.Det_a−Det_b=(a1−a2+2*wm)−(b1−b2−2*wm)=(a1−a2)−(b1−b2)+4*wm  (16)

In generally, related (but not necessarily immediately adjacent) frameswill have spatially correlated content. Because of the spatialcorrelation between the ‘a’ and ‘b’ frames, the image content is reducedwhile the watermark signal is reinforced. See equation 16.

For any one pair of frames selected by a watermark detector, thepolarity of the watermark could be either positive or negative. To allowfor this, the watermark detector may examine both polarities.

Watermark Embedding for Spot Colors

Product packaging is usually printed in one of two ways:

1. Process color printing using cyan, magenta yellow and/or black (CMYK)

2. Spot color printing (e.g., using special Pantone color or other inksets)

The majority of packaging is printed using spot colors mainly forreasons of cost and color consistency, and to achieve a wide color gamutover various packaging. Some conventional watermarking techniques embeddigital watermarks in either CMYK for printed images or RGB for digitalimages that are being displayed. But how to embed a watermark with aspot color?

An improvement addresses problem associated with watermarking spot colorimages. Preferably, packaging contains two (2) or more spot colors(e.g., printed cooperatively to achieve a certain color consistency).Each different color is altered to collectively carry a watermarksignal. A maximum signal strength within a user selectable visibilityconstraint with watermark in at least two (2) of the spot.

A maximized watermark signal is embedded preferably by modulating thespot color inks within a certain visibility constraint across the image.The approach models a color (ink) in terms of CIE Lab values. Lab is auniform perceptual color space where a unit difference in any colordirection corresponds to an equal perceptual difference.

The Lab axes are then scaled for the spatial frequency of the watermarkbeing added to the image, in a similar manner to the Spatial CieLabmodel by X. Zhang and B. A. Wandell, e.g., “A spatial extension ofCIELAB for digital color image reproduction,” in Proceedings of theSociety of Information Display Sumposium (SID '96), vol. 27, pp.731-734, San Jose, Calif., USA, June 1996. This is a uniform perceptualcolor space which we will call SLAB, where a unit difference in anycolor direction corresponds to an equal perceptual difference due to theaddition of a watermark signal at that spatial frequency.

The allowable visibility magnitude in SLAB is scaled by spatial maskingof the cover image. Spatial masking of the cover image can include thetechniques described by Watson in US Published Patent Application No. US2006-0165311 A1, which is hereby incorporated by reference in itsentirety, and can be used to scale the allowable visibility across theimage. This is a uniform perceptual color space which we will call VLAB,where the visibility circle is scaled to correspond to an equalperceptual difference due to the addition of a watermark signal at thatspatial frequency for that particular image.

The chrominance embedding techniques discussed above forms thefoundation for the present watermark embedding techniques. A relateddiscussion is found in U.S. patent application Ser. No. 13/975,919,filed Aug. 26, 2013, now U.S. Pat. No. 9,449,357, under the section“Chrominance watermark to embed using a full color visibility model,”which uses an iterative embed technique to insert a maximum watermarksignal into CMYK images.

The spot color technique described extends this work to embedding thatsupports special color inks (e.g., spot colors) used in packaging anduses a full color visibility model with spatial masking. A geometricenumerated embed approach can be used to evaluate a range of possibleink changes, which meet the user selected visibility constraint andpress constraints. The set of allowable ink changes are evaluated tochoose the pair of ink changes which result in the maximum signalstrength while meeting the visibility and press constraints.

FIG. 12 shows a detailed signal size view with ink increments of 2%, andthe addition of press constraints.

A user can insert a maximum watermark signal, while meeting anypre-required visibility constraint. The method has been applied to thecase of two spot colors and images have been produced which are morethan twice as robust to Gaussian noise as a single color image which isembedded using a luminance only watermark to the same visibility.

A method has been described which allows an image containing 2 or morespot colors to be embedded with a watermark in 2 of the spot colors,with the maximum signal strength within a user selectable visibilityconstraint.

A look-up table based approach can be used for given colors at givenlocations, and can easily be extended to 3 or more dimensions whilestill being computationally reasonable.

Additional related disclosure is found in U.S. patent application Ser.No. 13/975,919, now U.S. Pat. No. 9,449,357, under the heading sections“Geometric Enumerated Chrominance Watermark Embed for Spot Colors” and“Watermarking Embedding in Optimal Color Direction.”

Full-Color Visibility Model

A full color visibility model has been developed that uses separatecontrast sensitivity functions (CSFs) for contrast variations inluminance and chrominance (red-green and blue-yellow) channels. Thewidth of the CSF in each channel can be varied spatially depending onthe luminance of the local image content. The CSF can be adjusted sothat relatively more blurring occurs as the luminance of the localregion decreases. The difference between the contrast of the blurredoriginal and marked image can be measured using a color differencemetric.

This spatially varying CSF performed better than a fixed CSF in thevisibility model, approximating subjective measurements of a set of testcolor patches ranked by human observers for watermark visibility.

A full color visibility model can be a powerful tool to measurevisibility of an image watermark. Watermarks used for packaging can beinserted in the chrominance domain to obtain the best robustness perunit visibility. A chrominance image watermark is preferably embedded ina way that the color component in the cover image is minimally alteredand is hardly noticeable, due to human vision system's low sensitivityto color changes.

One example of a color visibility model is discussed relative to SpatialCIELAB (S-CIELAB). The accuracy of this model was tested by comparing itto human subjective tests on a set of watermarked color patches. Themodel was found to significantly overestimate the visibility of somedark color patches. A correction can be applied to the model for thevariation of the human contrast sensitivity function (CSF) withluminance. After luminance correction, better correlation was obtainedwith the subjective tests.

The luminance and chrominance CSF of the human visual system has beenmeasured for various retinal illumination levels. The luminance CSFvariation was measured by Van Nes (1967) and the chrominance CSFvariation by van der Horst (1969). These measurements show a variationin peak sensitivity of about a factor of 8 for luminance and 5 forchrominance over retinal illumination levels which change by about afactor of 100.

Since the retinal illumination can change by about a factor of 100between the lightest to darkest area on a page, the CSF peak sensitivityand shape can change significantly. The function is estimated by theaverage local luminance on the page, and a spatially dependent CSF isapplied to the image. This correction is similar to the luminancemasking in adaptive image dependent compression.

The luminance dependent CSF performed better than a fixed CSF in thevisibility model, when compared to subjective measurements of a set oftest color patches ranked by human observers for watermark visibility.In some cases, we use a method of applying a spatially dependent CSFwhich depends on local image luminance.

The visibility model can be used to embed watermark into images withequal visibility. During the embedding stage, the visibility model canpredict the visibility of the watermark signal and then adjust theembedding strength. The result will be an embedded image with a uniformwatermark signal visibility, with the embedding strength varyingdepending on the cover image's content.

The following documents are hereby incorporated herein by reference:Lyons, et al. “Geometric chrominance watermark embed for spot color,”Proc. Of SPIE, vol. 8664, Imaging and Printing in a Web 2.0 World IV,2013; Zhang et al. “A spatial extension of CIELAB for digitalcolor-image reproduction” Journal of the Society for Information Display5.1 (1997): 61-63; Van Nes et al. “Spatial modulation transfer in thehuman eye,” Journal of Optical Society of America, vol. 57, issue 3, pp.401-406, 1967; Van der Horst et al. “Spatiotemporal chromaticitydiscrimination,” Journal of Optical Society of America, vol. 59, issue11, 1969; and Watson, “DCTune,” Society for information display digestof technical papers XXIV, pp. 946-949, 1993.

In some cases, even better results can be achieved by combining anattention model with our above visibility model when embeddingwatermarks in color image data. An attention model generally predictswhere the human eye is drawn to when viewing an image. For example, theeye may seek out flesh tone colors and sharp contrast areas. One exampleattention model is described in Itti et al., “A Model of Saliency-BasedVisual Attention for Rapid Scene Analysis,” IEEE TRANSACTIONS ON PATTERNANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 11, NOVEMBER 1998, pgs.1254-1259, which is hereby incorporated herein by reference.

High visual traffic areas identified by the attention model, which wouldotherwise be embedded with a relatively strong or equal watermarksignal, can be avoided or minimized by a digital watermark embedder.

Additional related disclosure is found in Appendix D, attached andincluded as part of this specification, and which is hereby incorporatedherein by reference in its entirety.

Disclosure from Appendix D is also provided below:

Full-Color Visibility Model Using CSF Which Varies Spatially with LocalLuminance

Abstract:

A full color visibility model has been developed that uses separatecontrast sensitivity functions (CSFs) for contrast variations inluminance and chrominance (red-green and blue-yellow) channels. Thewidth of the CSF in each channel is varied spatially depending on theluminance of the local image content. The CSF is adjusted so that moreblurring occurs as the luminance of the local region decreases. Thedifference between the contrast of the blurred original and marked imageis measured using a color difference metric.

This spatially varying CSF performed better than a fixed CSF in thevisibility model, approximating subjective measurements of a set of testcolor patches ranked by human observers for watermark visibility. Theeffect of using the CIEDE2000 color difference metric compared toCIEDE1976 (i.e., a Euclidean distance in CIELAB) was also compared.

Introduction

A full color visibility model is a powerful tool to measure thevisibility of the image watermark. Image watermarking is a techniquethat covertly embeds additional information in a cover image, such thatthe ownership, copyright and other details about the cover image can becommunicated. Watermarks used for packaging are inserted in thechrominance domain to obtain the best robustness per unit visibility.See Robert Lyons, Alastair Reed and John Stach, “Geometric chrominancewatermark embed for spot color,” Proc. Of SPIE, vol. 8664, Imaging andPrinting in a Web 2.0 World IV, 2013. The chrominance image watermark isembedded in a way that the color component in the cover image isminimally altered and is hardly noticeable, due to human vision system'slow sensitivity to color changes.

This visibility model is similar to Spatial CIELAB (S-CIELAB). SeeXuemei Zhang and Brian A. Wandell, “A spatial extension of CIELAB fordigital color-image reproduction” Journal of the Society for InformationDisplay 5.1 (1997): 61-63. The accuracy of this model was tested bycomparing it to subjective tests on a set of watermarked color patches.The model was found to significantly overestimate the visibility of somedark color patches. A correction was applied to the model for thevariation of the human contrast sensitivity function (CSF) withluminance as described below. After luminance correction, goodcorrelation was obtained with the subjective tests.

The luminance and chrominance CSF of the human visual system has beenmeasured for various retinal illumination levels. The luminance CSFvariation was measured by Floris L. Van Nes and Maarten Bouman, “Spatialmodulation transfer in the human eye,” Journal of Optical Society ofAmerica, vol. 57, issue 3, pp. 401-406, 1967 and the chrominance CSFvariation by G J Van der Horst and Maarten Bouman, “Spatiotemporalchromaticity discrimination,” Journal of Optical Society of America,vol. 59, issue 11, 1969. These measurements show a variation in peaksensitivity of about a factor of 8 for luminance and 5 for chrominanceover retinal illumination levels which change by about a factor of 100.

Since the retinal illumination can change by about a factor of 100between the lightest to darkest area on a page, the CSF peak sensitivityand shape can change significantly. The function is estimated by theaverage local luminance on the page, and a spatially dependent CSF isapplied to the image. This correction is similar to the luminancemasking in adaptive image dependent compression. See G J Van der Horstand Maarten Bouman, “Spatiotemporal chromaticity discrimination,”Journal of Optical Society of America, vol. 59, issue 11, 1969.

The luminance dependent CSF performed better than a fixed CSF in thevisibility model, when compared to subjective measurements of a set oftest color patches ranked by human observers for watermark visibility.Results of our model with and without luminance correction are comparedto S-CIELAB in Section 2, Visual Model Comparison. The method ofapplying a spatially dependent CSF which depends on local imageluminance is described in Section 3, Pyramid Processing Method.

The visibility model is then used to embed watermark into images withequal visibility. During the embedding stage, the visibility model canpredict the visibility of the watermark signal and then adjust theembedding strength. The result will be an embedded image with a uniformwatermark signal visibility, with the embedding strength varyingdepending on the cover image's content. This method was compared to auniform strength embed in terms of both visibility and robustness, andthe results are shown in Section 4, Watermark Equal Visibility Embed.

Visual Model Comparison

Psychophysical Experiment

To test the full-color visibility model a psychophysical experiment wasconducted. The percept of degradation caused by the watermark wascompared to the results of the visibility model, as well as to theS-CIELAB metric.

A set of observers were asked to rate their perception of the imagedegradation of 20 color patch samples using a quality ruler. The qualityruler (illustrated in FIG. 13A) increases in watermark strength fromleft (B) to right (F). The color samples were viewed one at a time at aviewing distance of approximately 12 inches. The samples were presentedusing the Latin square design (see Geoffrey Keppel and Thomas Wickens,“Design and analysis: A researcher's handbook.” Prentice Hall, pp.381-386, 2004) to ensure a unique viewing order for each observer.

FIG. 13A shows quality ruler increasing in degradation from B (slight)to F (strong).

All 22 participants passed the Ishihara color test. There were eightfemale and 14 male participants, with an average age of 43. Theirprofessions and experience varied. Four people had never participated ina visibility experiment, 12 had some experience and six had participatedon several occasions.

Thumbnails of the 20 color patches are illustrated in FIG. 13B. Thecolor samples were chosen largely based on the results of a previousexperiment; where it was observed that the visibility model haddifficulty accurately predicting the observer response with darker colorpatches. Additionally, one color patch had a much higher perceived andpredicted degradation. Ten of the original samples were included in thesecond experiment. Dark patches, patches which were expected to have ahigher perception of degradation and memory colors were added tocomplete the set of 20 patches. The experiment and the quality rulerpatches were all printed with an Epson Stylus 4880 on Epson professionalphoto semi-gloss 16 inch paper.

FIG. 13B shows thumbnails of the 20 color patch samples with thewatermark applied.

The mean observer scores for the 20 color samples are plotted in FIG.14. In general the colors on the far right are lighter. As discussed inthe previous experiment, the cyan1 patch was observed to have a higherlevel of degradation. In this second experiment, other colors withsimilar properties were determined to have a similarly high perceptionof degradation.

FIG. 14 shows the mean observer responses with 95% confidence intervals.

Validation of the Visibility Model

The motivation for the psychophysical experiment is to test how well theproposed full-color visibility model correlates to the perception of thedegradation caused by the watermark signal. The model without and withthe luminance adjustment are plotted in FIG. 15 and FIG. 16,respectively.

FIG. 15 shows mean observer response compared with the proposedvisibility model. The solid black line is the polynomial trendline.

FIG. 16 shows mean observer response compared with the proposedvisibility model with luminance adjustment.

The addition of the luminance adjustment primarily affected the darkercolor patches, darkgreen, foliage and darkblue1. CIEDE94 and CIEDE2000color difference models were also considered, however there was not aclear advantage to using the more complex formulas.

FIG. 17 shows Mean observer response compared with S-CIELAB.

The S-CIELAB values are also plotted against the mean observer responseFIG. 17.

Two different methods were used to compare the different metrics to theobserver data, Pearson's correlation and the coefficient ofdetermination (R²). Both correlation techniques describe therelationship between the metric and observer scores. The coefficientindicates the relationship between two variables on a scale of +/−1, thecloser the values are to 1 the stronger the correlation is between theobjective metric and subjective observer results. The correlations aresummarized in Table 1.

TABLE 1 Table 1: Pearson and R² correlation between the observers' meanresponses and the objective metrics. For both tests, the proposedfull-color visibility model with the luminance adjustment shows thehighest correlation. Visibility model using CIE ΔE₇₆ No Adjust WithAdjust S-CIELAB Pearson 0.81 0.86 0.61 R² 0.70 0.85 0.38

As shown in Table 1, all three objective methods have a positivecorrelation to the subjective results with both correlation methods. Thefull-color visibility model with the luminance adjustment had thehighest correlation with both the Pearson and R² correlation tests,while S-CIELAB had the lowest.

Pyramid Processing Method

In image fidelity measures, the CSF is commonly used as a linear filterto normalize spatial frequencies such that they have perceptually equalcontrast thresholds. This can be described by the following shiftinvariant convolution:

$\begin{matrix}{{{\overset{\sim}{f}\left( {x,y} \right)} = {{{h\left( {x,y} \right)}*{f\left( {x,y} \right)}} = {\sum\limits_{m}\;{\sum\limits_{n}{{h\left( {m,n} \right)}{f\left( {{x - m},{y - n}} \right)}}}}}},} & (1)\end{matrix}$where f(x,y) is an input image, h(x,y) is the spatial domain CSF, and{tilde over (f)}(x,y) is the frequency normalized output image.

For our luminance dependent CSF model, we allow the CSF to varyspatially according to the local luminance of the image, i.e.:

$\begin{matrix}{{\overset{\sim}{f}\left( {x,y} \right)} = {\sum\limits_{m}\;{\sum\limits_{n}{{h\left( {m,{n;x},y} \right)}f{\left( {{x - m},{y - n}} \right).}}}}} & (2)\end{matrix}$

Since evaluating this shift variant convolution directly can becomputationally expensive, we seek an approximation that is moreefficient.

The use of image pyramids for fast image filtering is well-established.An image pyramid can be constructed as a set of low-pass filtered anddown-sampled images f_(l)(x,y), typically defined recursively asfollows:

$\begin{matrix}{{{f_{0}\left( {x,y} \right)} = {f\left( {x,y} \right)}}{and}} & (3) \\{{f_{l}\left( {x,y} \right)} = {\sum\limits_{m}\;{\sum\limits_{n}{{h_{0}\left( {m,n} \right)}{f_{l - 1}\left( {{{2\; x} - m},{{2\; y} - n}} \right)}}}}} & (4)\end{matrix}$for l>0 and generating kernel h₀(m, n). It is easily shown from thisdefinition that each level f_(l)(x,y) of an image pyramid can also beconstructed iteratively by convolving the input image with acorresponding effective kernel h_(l)(m,n) and down-sampling directly tothe resolution of the level, as follows:

$\begin{matrix}{{{f_{l}\left( {x,y} \right)} = {\sum\limits_{m}\;{\sum\limits_{n}{{h_{l}\left( {m,n} \right)}{f_{0}\left( {{{2^{l}\; x} - m},{{2^{l}\; y} - n}} \right)}}}}},} & (5)\end{matrix}$where h_(l)(m,n) is an l-repeated convolution of h₀(m,n) with itself.

For image filtering, the various levels of an image pyramid are used toconstruct basis images of a linear decomposition representing thepoint-spread response of the desired filtering, i.e.:

$\begin{matrix}{{{\overset{\sim}{f}\left( {x,y} \right)} = {\sum\limits_{l}\;{a_{l}{{\overset{\sim}{f}}_{l}\left( {x,y} \right)}}}},} & (6)\end{matrix}$where a_(l) is the coefficient of the basis function {tilde over(f)}_(l)(x, y) obtained by up-sampling the corresponding pyramid levelf_(l)(x,y) back to the base resolution.

We use the effective convolution kernel h_(l)(x,y) as an interpolatingkernel, i.e.,

$\begin{matrix}{{{{\overset{\sim}{f}}_{l}\left( {x,y} \right)} = {4^{l}{\sum\limits_{m}\;{\sum\limits_{n}{{h_{l}\left( {{x - {2^{l}m}},{y - {2^{l}n}}} \right)}{f_{l}\left( {m,n} \right)}}}}}},} & (7)\end{matrix}$such that each basis function {tilde over (f)}_(l)(x, y) can bedescribed by a simple shift-invariant convolution of the input imagewith a composite kernel {tilde over (h)}_(l)(x,y):{tilde over (f)} _(l)(x,y)={tilde over (h)} _(l)(x,y)*f(x,y),  (8)where {tilde over (h)}_(l)(x,y)=h_(l)(x,y)*h_(l)(x,y). Thus, consideringEq. (6), we assert that the optimal representation is obtained byminimizing the sum of the squared error between the desired CSF and theGaussian representation; i.e.,

$\begin{matrix}{{a = {\arg\mspace{11mu}{\min\limits_{a}E}}},{where}} & (8) \\{{E = {\sum\limits_{x}\;{\sum\limits_{y}\left( {{h\left( {x,y} \right)} - {\sum\limits_{l}{a_{l}{{\overset{\sim}{h}}_{l}\left( {x,y} \right)}}}} \right)^{2}}}},} & (9)\end{matrix}$and a=[a₁, a₂, . . . ]. This is a standard linear least-squares problemand can be solved using standard software packages, like Matlab® or GNUOctave. Further, the optimization can be pre-calculated for each localluminance of interest and stored in a look-up table, noting that for ourapplication each coefficient a₁ is spatially varying according to thelocal luminance level L_(f)=L_(f)(x,y) of f(x,y), i.e.,a _(l) =a _(l)(L _(f))=a _(l)(L _(f)(x,y)).

While the development of our approach has been conducted for basis imageat the resolution of the input image, the procedure can be conductedwithin a multi-resolution scheme, reducing the calculation of thespatially variant convolution in Eq. (3.2) into a pyramid reconstructionwith spatially variant analysis coefficients.

Watermark Equal Visibility Embed

FIG. 18 shows an example from a cover image mimicking a package design.The design has two embedding schemes: on the left the watermark signalstrength is uniform across the whole image, and on the right thewatermark signal strength is adjusted based on the prediction from thevisibility model. Since the human visual system is approximately a peakerror detector, the image degradation caused by the watermark signal isdetermined by the most noticeable area. In this example, the hilly areain the background has the most noticeable degradation, as shown in themagnified insets. The visibility model is used to find this severedegradation. The signal strength in this area is reduced which improvesthe overall visibility of the embedded image, making it more acceptable.The total watermark signal on the right is 40% more than that on theleft, but visually, the marked image on the right is preferable to theleft one, because the degradation in the most noticeable area is reducedsignificantly.

FIG. 19 shows the calculated visibility for the uniform signal strengthembedding (left) and the visibility model adjusted embedding (right).Notice that the visibility map is smoother on the right than on theleft.

In terms of watermark detection, the embedding scheme with visibilitymodel based adjustment can accommodate more watermark signal withoutcreating a very noticeable degradation, thus making the detection morerobust. To demonstrate the powerfulness of applying the visibilitymodel, we performed a stress test with captures of 4 images from the twoembedding schemes at various distances and perspectives. The other 3images from the uniform visibility embedding are shown in FIG. 20. Theirvisibility maps are not included but instead the standard deviation ofeach visibility map is listed in Table 2. The percentage of successfuldetection is shown in Table 3.

These two tables show that the equal visibility embedding showed asignificant visibility improvement over the uniform strength embeddingscheme, together with robustness that was about the same or better.

FIG. 18 shows watermark embedding with uniform signal strength (left)and equal visibility from the visibility model (right). The insets aremagnified to show image detail.

FIG. 19 shows visibility map from uniform signal strength embedding(left) and equal visibility embedding (right).

FIG. 20 shows Apple tart, Giraffe stack and Pizza puff design used intests.

Table 2 shows standard deviation of the visibility maps on the 4 imagesfrom the two embedding schemes.

TABLE 2 Test image Uniform strength embedding Equal visibility embeddingGranola 18.32 9.71 Apple Tart 8.19 4.96 Giraffe Stack 16.89 11.91 PizzaPuff 11.81 8.27

Table 3 shows detection rate on 4 images from the two embedding schemes,out of 1000 captures each image/embedding.

TABLE 3 Test image Uniform strength embedding Equal visibility embeddingGranola 18% 47% Apple Tart 50% 58% Giraffe Stack 47% 49% Pizza Puff 63%61%Conclusions

A full color visibility model has been developed which has goodcorrelation to subjective visibility tests for color patches degradedwith a watermark. The best correlation was achieved with a model thatapplied a luminance correction to the CSF.

The model was applied during the watermark embed process, using apyramid based method, to obtain equal visibility. Better robustness andvisibility was obtained with equal visibility embed than uniformstrength embed.

Discussion

One goal of a color visibility model is to create an objective visualdegradation model due to digital watermarking of an image. For example,a model may predict how noticeable or visible image changes will be dueto watermark insertion. Highly noticeable changes can be reduced ormodified to reduce watermark visibility, and/or to create equalwatermark visibility (or lack thereof) across an image. For example, anerror metric above or relative to the standard “Just NoticeableDifference” (JND) can be used to determine noticeable changes.

In a first implementation, with reference to FIG. 21, a digitalwatermarked image is compared to a processed version of an originalimage to determine a visibility map. The visibility map can be used toweight watermark embedding of the original image, e.g., to reducewatermark strength in high visibility areas. The process starts withconversion of an original image into the so-call CIELAB space, resultingin L*, a* and b* color representations. As mentioned above, the L*coordinate represents the perceived lightness or luminance, an L* valueof 0 indicates black and a value of 100 indicates white. The CIE a*coordinate position goes between “redness” (positive) and “greenness”(negative), while the CIE b* goes between “yellowness” (positive) and“blueness” (negative). The original image is digitally watermarked andthen converted into the CIELAB space. For example, the watermarking forthis initial process may use a uniform embedding strength across theentire image.

Contrast between the original image and the marked image can bedetermined, and then contrast sensitivity functions (CSFs) can beapplied to each of the L*, a* and b* channels. For example, the L* CSFsdiscussed in Daly, “Visible differences predictor: an algorithm for theassessment of image fidelity,” F. L. van Nes et al. “Spatial ModulationTransfer in the Human Eye,” J. Opt. Soc. Am., Vol. 57, Issue 3, pp.401-406 (1967), or Johnson et al, “On Contrast Sensitivity in an ImageDifference Model,” PICS 2002: Image Processing, Image Quality, ImageCapture Systems Conference, Portland, Oreg., April 2002; p. 18-23 (whichis herein incorporated herein in its entirety), can be used. In othercases a bandpass filter, with a drop off toward low-frequencies, can beapplied to the L*. The processed or blurred L* channel (from theoriginal image) can be used to determine visibility masking. Forexample, areas of high contrast, edges, features, high variance areas,can be identified for inclusion of more or less watermarking strength.Some areas (e.g., flat area, edges, etc.) can be entirely masked out toavoid watermarking all together.

For the a* and b* channels, chrominance CSFs can be applied to therespective channels, e.g., such CSFs as discussed in Johnson et al,“Darwinism of Color Image Difference Models;” G. J. C. van der Horst etal., “Spatiotemporal chromaticity discrimination,” J. Opt. Soc. Am.,59(11), 1482-1488, 1969; E. M. Granger et al., “Visual chromaticitymodulation transfer function,” J. Opt. Soc. Am., 63(9), 73-74, 1973; K.T. Mullen, “The contrast sensitivity of human colour vision to red-greenand blue-yellow chromatic gratings,” J. Physiol., 359, 381-400, 1985;each of which are hereby incorporated herein by reference in theirentirety. In other cases, a low-pass filter is used which has a lowercut-off frequency relative to the CSF of luminance.

Channel error difference can then be determined or calculated. Forexample, on a per pixel basis, L*, a* and b* data from the originalimage are compared to the blurred (e.g., processed with respective CSFs)L*, a* and b*channels from the watermarked image. One comparisonutilizes ΔE₇₆:

-   Using (L*₁, a*₁, b*₁) and (L*₂, a*₂, b*₂), two colors in L*a*b*, the    error between two corresponding pixel values is:

ΔE*_(ab)=√{square root over (L*₂−L*₁)²+(a*₂−a*₁)²+(b*₂−b*₁)²)}, whereΔE*_(ab)≈2.3 corresponds to a JND (just noticeable difference). Othercomparisons may utilize, e.g., ΔE₉₄ or ΔE₂₀₀₀.

Of course, and more preferably used, is an error determination for theblurred (CSF processed) L*a*b* from the original image and the CSFblurred L*a*b* from the watermarked image.

The output of the Calculate Channel Difference module identifies errormetrics. The error metrics can be used to identify image areas likely toinclude high visibility due to the inserted digital watermark signal. Wesometimes refer to this output as an “error map”. Typically, the lowerthe error, the less visible the watermark is at a particular area, imageblocks or even down to a signal pixel.

The visibility mask and the error map can be cooperatively utilized toguide digital watermarking. For example, watermark signal gain can bevaried locally according to the error map, and areas not conducive toreceive digital watermark, as identified in the visibility mask, canaltogether be avoided or receive a further signal reduction.

One limitation of the FIG. 21 model is that it does not take intoaccount local luminance influences for Contrast Sensitivity Functions(CSF), particularly for the a* and b* chrominance channels. Withreference to FIG. 22, we propose a color visibility model for use with adigital watermark embedder that seeks equal visibility across an imageby locally varying watermarking embedding strength based on predictedvisibility influenced, e.g., by local image luminance. A CSF for eachcolor channel can be varied spatially depending on the luminance of thelocal image content.

The luminance content of the original image provides potential maskingof changes due to watermarking in chrominance as well as luminance. Forexample, where a watermark signal comprises mostly high frequencycomponents, the masking potential of the original image is greater atregions with high frequency content. We observe that most high frequencycontent in a typical host image is in the luminance channel. Thus, theluminance content of the host is the dominant contributor to maskingpotential for luminance changes and chrominance changes for highfrequency components of the watermark signal.

Returning to FIG. 22, we may add several modules relative to the FIG. 21system, e.g., “Calculate Local Luminance” and “blur SCALED CSF” modules.The FIG. 22 visibility model system uses separate CSFs for contrastvariations in luminance and chrominance (red-green and blue-yellow)channels. The width, characteristics or curve of the CSF in each channelcan be scaled or modified depending on the luminance of the local imagecontent. For example, for a given pixel, local luminance in aneighborhood around the pixel can be evaluated to determine a localbrightness value. The local brightness value can be used to scale ormodified a CSF curve. The neighborhood may include, e.g., 4, 8 or morepixels. In some cases, the CSF is adjusted so that more blurring occursas the luminance of the local region decreases. The error differencebetween the contrast of the blurred (or unblurred) original and theblurred marked image can be measured using a color difference metric,e.g., ΔE76, ΔE94 or ΔE2000.

With reference to FIG. 23A, one objective may include embedding digitalwatermarking into images with equal visibility. That is, the imageincludes watermarking embedded therein at different signal strengthvalues to achieve uniform or equal visibility. During the embeddingstage, the visibility model can predict the visibility of the watermarksignal and then adjust the embedding strength. The result will be anembedded image with a uniform watermark signal visibility, with theembedding strength varying locally across the image depending oncharacteristics of the cover image's content. For example, a visibilitymap generated from the FIG. 22 system is used to reshape (e.g., locallyscale according to an error map and/or mask embedding or avoidance areasaccording to a visibility map) a watermark signal. The original signalis then embedded with the reshaped watermark signal to create an equalvisibility embedded (EVE) image. In such a case, the watermark signallocally varies to achieve an overall equal visibility.

Some visibility advantages of EVE vs. uniform strength embedding (USE)are shown in FIG. 23B. The visibility of the USE varies from area toarea, as see in the bottom left image. In comparison, when embedding thesame image area with EVE (bottom right image), the watermark visibilityappears equal. The bottom left and right images represent the same imagearea highlighted in blue in the upper right image.

CONCLUDING REMARKS

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicant hereby incorporates by reference each of theabove referenced patent documents in its entirety.

The methods, processes, components, apparatus and systems describedabove may be implemented in hardware, software or a combination ofhardware and software. For example, the watermark encoding processes andembedders may be implemented in software, firmware, hardware,combinations of software, firmware and hardware, a programmablecomputer, electronic processing circuitry, with a processor, parallelprocessors or other multi-processor configurations, and/or by executingsoftware or instructions with one or more processors or dedicatedcircuitry. Similarly, watermark data decoding or decoders may beimplemented in software, firmware, hardware, combinations of software,firmware and hardware, a programmable computer, electronic processingcircuitry, and/or by executing software or instructions with aprocessor, parallel processors or other multi-processor configurations.

The methods and processes described above (e.g., watermark embedders anddetectors) also may be implemented in software programs (e.g., writtenin C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme, Ruby,executable binary files, etc.) stored in memory (e.g., a computerreadable medium, such as an electronic, optical or magnetic storagedevice) and executed by a processor (or electronic processing circuitry,hardware, digital circuit, etc.).

While one embodiment discusses inverting the polarity in a second colorchannel (e.g., a ‘b’ channel), one could also invert the polarity in thefirst color channel (e.g., an ‘a’ channel) instead. In such a case, thefirst color channel is then preferably subtracted from the second colorchannel.

The particular combinations of elements and features in theabove-detailed embodiments (including Appendix D) are exemplary only;the interchanging and substitution of these teachings with otherteachings in this and the incorporated-by-reference patent documents arealso contemplated.

What is claimed is:
 1. An apparatus comprising: memory storing: i) aluminance contrast sensitivity function (CSF1), ii) a chrominancecontrast sensitivity function (CSF2), and iii) data representing colorimagery; means for estimating degradation of image areas associated withan application of signal encoding in the data representing color imageryby applying the CSF1 and the CSF2 to the data representing colorimagery, in which the CSF1 varies depending on luminance valuesassociated with local regions of the data representing color imagery,and in which the CSF1 is used for processing luminance data and the CSF2is used for processing chrominance data; and means for changing the datarepresenting color imagery with signal encoding, in which the signalencoding is guided based on results obtained from said means forestimating including estimated degradation of image areas.
 2. Theapparatus of claim 1 in which the CSF1 varies spatially.
 3. Theapparatus of claim 2 in which the CSF2 varies spatially in terms ofspatial width.
 4. The apparatus of claim 1 in which the CSF1 variesspatially in terms of spatial width.
 5. The apparatus of claim 1 inwhich the means for estimating degradation produces image blurring asthe estimated degradation, in which the CSF1 varies so that relativelymore blurring occurs as luminance of a local image region decreases. 6.The apparatus of claim 1 in which the means for changing utilizesresults obtained from the means for estimating by varying signalencoding strength across different image areas of the data representingcolor imagery based on estimated degradation of the different imageareas.
 7. The apparatus of claim 6 in which estimated degradation of thesignal encoding across the different image areas comprises uniformestimated degradation.
 8. The apparatus of claim 1 further comprisingmeans for applying an attention model to the data representing colorimagery to predict visual traffic areas.
 9. The apparatus of claim 8 inwhich the means for changing utilizes predicted visual traffic areas andthe estimated degradation of image areas.
 10. The apparatus of claim 1in which the chrominance contrast sensitivity function (CSF2) comprisesa blue-yellow contrast sensitivity function and a red-green contrastsensitivity function.
 11. The apparatus of claim 1 in which the CSF2varies depending on luminance values associated with local regions ofthe obtained color image data.
 12. The apparatus of claim 1 in whichsaid means for changing the data representing color imagery with signalencoding encodes a payload into the data representing color imagery. 13.The apparatus of claim 1 in which the color imagery comprises video. 14.A method comprising: obtaining color image data; changing the colorimage data with signal encoding, the signal encoding comprising apayload, said changing yielding encoded color image data; comparing theencoded color image data to the color image data to determine avisibility map, the visibility map comprising a luminance contrastsensitivity function (CSF1) and a chrominance contrast sensitivityfunction (CSF2); weighting the signal encoding per the visibility map sothat local image areas within the color image data are weighteddifferently, said weighting yielding weighted signal encoding; encodingthe color image data with the weighted signal encoding to yield locallyvaried encoded color image data.
 15. The method of claim 14 in which theCSF1 introduces image blurring, and in which the CSF1 varies so thatrelatively more blurring occurs as luminance of a local image regiondecreases.
 16. The method of claim 15 in which said weighting variessignal encoding strength across different local image regions of thecolor image data to yield uniform visibility of the signal encodingacross the color image data.
 17. The method of claim 16 in which thecolor image data represents video data.
 18. The method of claim 14 inwhich the CSF1 varies spatially in terms of spatial width.
 19. Themethod of claim 14 in which the CSF2 varies spatially in terms ofspatial width.
 20. The method of claim 14 in which the CSF2 comprises ablue-yellow contrast sensitivity function and a red-green contrastsensitivity function.
 21. A non-transitory computer readable mediumcomprising instructions, which when executed configure one or moreprocessors to: access color image data; change the color image data withsignal encoding, the signal encoding comprising a payload, said changingyielding encoded color image data; compare the encoded color image datato the color image data to determine a visibility map, the visibilitymap comprising a luminance contrast sensitivity function (CSF1) and achrominance contrast sensitivity function (CSF2); weight the signalencoding per the visibility map so that local image areas within thecolor image data are weighted differently to yield weighted signalencoding; encode the color image data with the weighted signal encodingto yield locally varied encoded color image data.
 22. The non-transitorycomputer readable medium of claim 21 in which the CSF1 introduces imageblurring, and in which the CSF1 varies so that relatively more blurringoccurs as luminance of a local image region decreases.
 23. Thenon-transitory computer readable medium of claim 22 in which the signalencoding strength varies across different local image regions of thecolor image data to yield uniform visibility of the signal encodingacross the color image data.
 24. The non-transitory computer readablemedium of claim 21 in which the CSF1 varies spatially in terms ofspatial width.
 25. The non-transitory computer readable medium of claim21 in which the CSF2 varies spatially in terms of spatial width.
 26. Thenon-transitory computer readable medium of claim 21 in which the CSF2comprises a blue-yellow contrast sensitivity function and a red-greencontrast sensitivity function.
 27. The non-transitory computer readablemedium of claim 21 in which the color image data represents video data.